{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# hello_milvus Demo\n",
    "\n",
    "    hello_milvus.ipynb demonstrates the basic operations of PyMilvus, a Python SDK of Milvus.\n",
    "    Before running, make sure that you have a running Milvus instance.\n",
    "\n",
    "1. connect to Milvus\n",
    "2. create collection\n",
    "3. insert data\n",
    "4. create index\n",
    "5. search, query, and hybrid search on entities\n",
    "6. delete entities by PK\n",
    "7. drop collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "\n",
    "from pymilvus import (\n",
    "    connections,\n",
    "    utility,\n",
    "    FieldSchema, CollectionSchema, DataType,\n",
    "    Collection,\n",
    ")\n",
    "\n",
    "fmt = \"\\n=== {:30} ===\\n\"\n",
    "search_latency_fmt = \"search latency = {:.4f}s\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. connect to Milvus\n",
    "\n",
    "Add a new connection alias `default` for Milvus server in `localhost:19530`. Actually the \"default\" alias is a buildin in PyMilvus. If the address of Milvus is the same as `localhost:19530`, you can omit all\n",
    "parameters and call the method as: `connections.connect()`.\n",
    "\n",
    "Note: the `using` parameter of the following methods is default to \"default\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does collection hello_milvus exist in Milvus: False\n"
     ]
    }
   ],
   "source": [
    "connections.connect(\"default\", host=\"localhost\", port=\"19530\")\n",
    "\n",
    "has = utility.has_collection(\"hello_milvus\")\n",
    "print(f\"Does collection hello_milvus exist in Milvus: {has}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. create collection\n",
    "We're going to create a collection with 3 fields.\n",
    "\n",
    "| | field name | field type | other attributes |       field description      |\n",
    "|---| :--------: | :----------: | :----------------: | :----------------------------: |\n",
    "|1|    \"pk\"    |    Int64   |  is_primary=True, auto_id=False |      \"primary field\"         |\n",
    "|2|  \"random\"  |    Double  |                  |      \"a double field\"        |\n",
    "|3|\"embeddings\"| FloatVector|     dim=8        |  \"float vector with dim 8\"   |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "fields = [\n",
    "    FieldSchema(name=\"pk\", dtype=DataType.INT64, is_primary=True, auto_id=False),\n",
    "    FieldSchema(name=\"random\", dtype=DataType.DOUBLE),\n",
    "    FieldSchema(name=\"embeddings\", dtype=DataType.FLOAT_VECTOR, dim=8)\n",
    "]\n",
    "\n",
    "schema = CollectionSchema(fields, \"hello_milvus is the simplest demo to introduce the APIs\")\n",
    "\n",
    "hello_milvus = Collection(\"hello_milvus\", schema)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. insert data\n",
    "\n",
    "We are going to insert 3000 rows of data into `hello_milvus`. Data to be inserted must be organized in fields.\n",
    "\n",
    "The insert() method returns:\n",
    "- either automatically generated primary keys by Milvus if auto_id=True in the schema;\n",
    "- or the existing primary key field from the entities if auto_id=False in the schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entities in Milvus: 3000\n"
     ]
    }
   ],
   "source": [
    "entities = [\n",
    "    # provide the pk field because `auto_id` is set to False\n",
    "    [i for i in range(3000)],\n",
    "    [float(random.randrange(-20, -10)) for _ in range(3000)],  # field random\n",
    "    [[random.random() for _ in range(8)] for _ in range(3000)],  # field embeddings\n",
    "]\n",
    "\n",
    "insert_result = hello_milvus.insert(entities)\n",
    "\n",
    "print(f\"Number of entities in Milvus: {hello_milvus.num_entities}\")  # check the num_entites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. create index\n",
    "We are going to create an IVF_FLAT index for hello_milvus collection.\n",
    "\n",
    "create_index() can only be applied to `FloatVector` and `BinaryVector` fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Status(code=0, message='')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = {\n",
    "    \"index_type\": \"IVF_FLAT\",\n",
    "    \"metric_type\": \"L2\",\n",
    "    \"params\": {\"nlist\": 128},\n",
    "}\n",
    "\n",
    "hello_milvus.create_index(\"embeddings\", index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. search, query, and hybrid search\n",
    "After data were inserted into Milvus and indexed, you can perform:\n",
    "- search based on vector similarity\n",
    "- query based on scalar filtering(boolean, int, etc.)\n",
    "- hybrid search based on vector similarity and scalar filtering.\n",
    "\n",
    "Before conducting a search or a query, you need to load the data in `hello_milvus` into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Start searching based on vector similarity ===\n",
      "\n",
      "hit: (distance: 0.0, id: 2998), random field: -20.0\n",
      "hit: (distance: 0.1339951753616333, id: 1871), random field: -13.0\n",
      "hit: (distance: 0.16615478694438934, id: 1180), random field: -16.0\n",
      "hit: (distance: 0.0, id: 2999), random field: -16.0\n",
      "hit: (distance: 0.10607236623764038, id: 764), random field: -11.0\n",
      "hit: (distance: 0.14412546157836914, id: 750), random field: -11.0\n",
      "search latency = 0.3159s\n",
      "\n",
      "=== Start querying with `random > -14` ===\n",
      "\n",
      "query result:\n",
      "-{'pk': 0, 'random': -13.0, 'embeddings': [0.07525, 0.534547, 0.778204, 0.646336, 0.800183, 0.998726, 0.545411, 0.631751]}\n",
      "search latency = 0.2571s\n",
      "\n",
      "=== Start hybrid searching with `random > -12` ===\n",
      "\n",
      "hit: (distance: 0.3116421699523926, id: 801), random field: -11.0\n",
      "hit: (distance: 0.34958416223526, id: 568), random field: -11.0\n",
      "hit: (distance: 0.3618723750114441, id: 1105), random field: -11.0\n",
      "hit: (distance: 0.10607236623764038, id: 764), random field: -11.0\n",
      "hit: (distance: 0.14412546157836914, id: 750), random field: -11.0\n",
      "hit: (distance: 0.29973354935646057, id: 2716), random field: -11.0\n",
      "search latency = 0.1434s\n"
     ]
    }
   ],
   "source": [
    "hello_milvus.load()\n",
    "\n",
    "# search based on vector similarity\n",
    "print(fmt.format(\"Start searching based on vector similarity\"))\n",
    "vectors_to_search = entities[-1][-2:]\n",
    "search_params = {\n",
    "    \"metric_type\": \"l2\",\n",
    "    \"params\": {\"nprobe\": 10},\n",
    "}\n",
    "\n",
    "start_time = time.time()\n",
    "result = hello_milvus.search(vectors_to_search, \"embeddings\", search_params, limit=3, output_fields=[\"random\"])\n",
    "end_time = time.time()\n",
    "\n",
    "for hits in result:\n",
    "    for hit in hits:\n",
    "        print(f\"hit: {hit}, random field: {hit.entity.get('random')}\")\n",
    "print(search_latency_fmt.format(end_time - start_time))\n",
    "\n",
    "# -----------------------------------------------------------------------------\n",
    "# query based on scalar filtering(boolean, int, etc.)\n",
    "print(fmt.format(\"Start querying with `random > -14`\"))\n",
    "\n",
    "start_time = time.time()\n",
    "result = hello_milvus.query(expr=\"random > -14\", output_fields=[\"random\", \"embeddings\"])\n",
    "end_time = time.time()\n",
    "\n",
    "print(f\"query result:\\n-{result[0]}\")\n",
    "print(search_latency_fmt.format(end_time - start_time))\n",
    "\n",
    "# -----------------------------------------------------------------------------\n",
    "# hybrid search\n",
    "print(fmt.format(\"Start hybrid searching with `random > -12`\"))\n",
    "\n",
    "start_time = time.time()\n",
    "result = hello_milvus.search(vectors_to_search, \"embeddings\", search_params, limit=3, expr=\"random > -12\", output_fields=[\"random\"])\n",
    "end_time = time.time()\n",
    "\n",
    "for hits in result:\n",
    "    for hit in hits:\n",
    "        print(f\"hit: {hit}, random field: {hit.entity.get('random')}\")\n",
    "print(search_latency_fmt.format(end_time - start_time))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. delete entities by PK\n",
    "You can delete entities by their PK values using boolean expressions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query before delete by expr=`pk in [2, 3]` -> result: \n",
      "-{'pk': 2, 'random': -14.0, 'embeddings': [0.976175, 0.088528, 0.806287, 0.004207, 0.30336, 0.298667, 0.279592, 0.421679]}\n",
      "-{'pk': 3, 'random': -20.0, 'embeddings': [0.230225, 0.149853, 0.704977, 0.938874, 0.092708, 0.104514, 0.839864, 0.235236]}\n",
      "\n",
      "query after delete by expr=`pk in [2, 3]` -> result: []\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ids = insert_result.primary_keys\n",
    "expr = f\"pk in [{ids[2]}, {ids[3]}]\"\n",
    "\n",
    "result = hello_milvus.query(expr=expr, output_fields=[\"random\", \"embeddings\"])\n",
    "print(f\"query before delete by expr=`{expr}` -> result: \\n-{result[0]}\\n-{result[1]}\\n\")\n",
    "\n",
    "hello_milvus.delete(expr)\n",
    "\n",
    "result = hello_milvus.query(expr=expr, output_fields=[\"random\", \"embeddings\"])\n",
    "print(f\"query after delete by expr=`{expr}` -> result: {result}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. drop collection\n",
    "Finally, drop the hello_milvus collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "utility.drop_collection(\"hello_milvus\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "08729c1b09183f4f38e81bb10929f98dd3c2fb886eeaa68ef9ddc9d2071f5c86"
  },
  "kernelspec": {
   "display_name": "Python 3.7.0 64-bit ('pymilvus': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
